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1. INTRODUCTION 


An experiment was designed to test the ability of the Classifica- 
tion and Mensuration Subsystem (CAMS) rework operations to improve 
wheat proportion estimates for segments that had been processed 
previously. Eleven segments were selected for the experiment, 
including three in Kansas and three in Texas, with the remaining 
five segments distributed in Montana and North and South Dakota. 
The Kansas and Texas sites were selected to provide information 
on the southern Great Plains. The remaining sites were selected 
to augment the knowledge acquired from the blind site study of 
the mixed and spring wheat sites in the northern Great Plains. 

The acquisition dates were selected to be representative of 
imagery available in actual operations . No more than one acqui- 
sition per biophase would be used, and biophases were to be deter- 
mined by actual crop calendars. All sites were intensive test 
sites (ITS's) over which at least two passes had been made , and 
each had an acquisition from either biophase 2 or 3. 

All sites were worked by each of four Analyst-Interpreter/Data 
Processing Analyst (AI/DPA) Teams randomly selected from teams 
which were familiar with CAMS rework methodology. Each AI/DPA 
Team reviewed the initial processing of each segment and accepted 
or reworked it for an estimate of the proportion of small grains 
in the segment. Specific procedures and required rework products 
are described in section 2. The classification results are given 
in table I; table II lists the acquisitions; and tables III and 
IV describe classification errors and performance results between 
CAMS regular and ITS rework. 

The experimental design provided 44 classification runs with a 
factor of 4 (AI/DPA Teams) times 11 (sites) . However, two of the 


sites were declared unworkable by some of the teams and were 
subsequently dropped from the experiment. 

2 . PROCEDURES 

a. The AI/DPA Team reviews processed acquisitions for a segment 
and determines whether the processed acquisition acreage 
estimation is acceptable. 

b. The AI/DPA Team determines if rework is necessary. 

c. If rework is not necessary, the AI/DPA Team confers on the 
results and forwards the data to the Accuracy Assessment 
Team. 

d. If rework is necessary, the AI/DPA Team confers on the rework 
technique . 

e. The segment is reworked by an AI/DPA Team, using the follow- 
ing rework products : 

• All classification maps 

• All evaluation forms 

• All ancillary data 

• Color film for all classified acquisitions 

• Batch printouts 

The results are discussed and forwarded to the Accuracy 
Assessment Team. 

f. As a ground rule, selected acquisitions are not discussed 
between the teams. 

g. Team arrangement: 

« One AI/DPA interpretation is selected randomly from AI/DPA 
Teams A, B, C, or D. 

• Four AI/DPA Teams are selected randomly from all experienced 
AI/DPA Teams. 
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3. DATA ANALYSIS 


Since each AI/DPA Team was provided with CAMS classification 
results from regular processing of each segment, the CAMS rework 
proportion estimates are expected to be correlated with the 
initial estimates. 

To study classification errors of the CAMS rework procedure and 
to determine whether or not proportion estimates are improved, a 
linear regression was performed using the CAMS rework estimate as 
the dependent variable and the CAMS regular estimate as an inde- 
pendent variable. The arc sin /p transformation, where p denotes 
proportion, was employed to stabilize the variance and thus 
enable valid statistical inferences to be made. Details on 
terminology and results are presented in section 3.2. The 
regression is significant, and 44 percent of the total errors 
in the CAMS rework estimates are caused by errors in the CAMS 
regular estimates. 

Elimination of the effect of CAMS regular outputs on the CAMS 
rework results was considered, and residuals were obtained. An 
analysis of variance (ANOVA) was performed on the residuals in 
order to study whether or not the CAMS rework performance varied 
according to AI/DPA Team and/or site. These results are given in 
section 3.3. No significant difference is detected between the 
four AI/DPA Teams. However, a difference is observed between 
sites but is significant only at the 10-percent level. Since 
this ANOVA is for a 4-by-9 factorial design with no replications, 
the interaction between sites and teams cannot be evaluated and 
hence cannot be tested for its statistical significance. 
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3.1 ANALYSIS OF ERROR SOURCES 


An analysis of error sources by segment is set out below. 

i 

• Segment 1687, Hand County (1) , South Dakota — Only biophases 1 
and 3 were available for rework. Biophase 3 was acquired in 
late July, which, according to the crop calendar, was the time 
for harvest and should probably be called biophase 4. Some 
grass in this biophase has the signature of ripening wheat. 
Therefore, labeling errors occurred, which resulted in 

o veres timation . 

• Segment 1960, Finney County, Kansas — Some of the field numbers 
used in the fall-planted wheat inventory report could not be 
located on the updated field overlay. This indicates that some 
wheatfield boundaries were changed later in the growing season. 
Thus, the original small-grain proportions, which were computed 
on the basis of the fall inventory report, were inadequate. As 
a result, the ground-truth small-grain proportions presented in 
table I may be in error. The results for this segment can be 
interpreted only with reservations. 

Some of the wheatfields in this segment were in very poor con- 
dition and did not have a wheat signature. For this reason, 
they were not recognized by the interpreters as wheat and were 
classified as nonwheat. A threshold of 8.6 percent occurred 
in the ground-truth area. According to the ground- truth report, 
a major portion of the thresholded area is actually wheat, 
which indicates a wheat signature was missed. 

• Segment 1962, Saline County, Kansas — The ground-truth small- 
grain proportions were computed based on the fall-planted wheat 
inventory report. No final inventory was available for this 
segment. 

AI/DPA Teams C and D missed some wheat signatures, which caused 
the underestimation. Both AI/DPA Teams A and B had estimates 
which were slightly higher than the ground truth; a very thorough 
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comparison of the classification map with the ground truth 
indicates that both are good classifications. 

• Segment 1963, Rice County, Kansas — This segment contains 
narrow strip fields, which are less than one pixel in width 
and for which no training fields were provided. Most of the 
pixels in these fields were classified as mixed. It was also 
found that most of these mixed pixels were actually wheat but 
misclassif ied as nonwheat, resulting in a reduction of at 
least 6 percent in wheat from the estimated small-grain 
proportions . 

In addition to the problems caused by the narrow strip fields, 
labeling errors were made by AI/DPA Teams C and D. 

• Segment 1965, Burke County, North Dakota — Every team under- 
estimated the small-grain proportions in this segment. Some 
wheat signatures were mislabeled as nonwheat. Only biophases 1 
and 2 were available for rework. Use of a later biophase acqui- 
sition would made it possible to identify these signatures 
correctly . 

• Segment 1969, Toole County, Montana — The classification results 
were generally good. AI/DPA Team C mislabeled one nonwheat 
signature, which probably caused overestimation. 

• Segment 1970, Liberty County, Montana — As was the case with 
segment 1963, problems occurred because of narrow strip fields, 
resulting in the misclassif ication of mixed pixels. 

• Segment 1978, Randall County, Texas — Small-grain proportions 
were originally underestimated at least 25 percent by the AI/DPA 
Teams. The major problem was volunteer wheat which was counted 
as small grain. These fields of volunteer wheat were plowed 
when the data were acquired. The ground- truth proportion was 
corrected to exclude these fields from the small grain. The 
ground-truth proportion is therefore reduced 13 percent. 

Missing wheat signatures were also an error source. Signatures 
were mislabeled because of the low ground cover (60 to 79 percent) . 
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• Segment 1979 , Deaf Smith County, Texas — All the volunteer 
wheatfields in this segment were plowed when data were acquired. 
After the ground- truth proportions were adjusted to exclude 

the volunteer wheat, they were almost identical to the classi- 
cation result. 

• Segment 1980, Oldham County, Texas — All the volunteer wheat- 
fields were listed in the crop inventory report as summer fallow. 
Original ground-truth proportions were adequately computed, 

• Segment 1986, Hand County (2), South Dakota — Since clouds and 
haze were observed in biophases 2 and 3, only biophase 1 was 
considered workable. Because of the poor data, this segment 
should not have been worked or reworked. 


3.2 REGRESSION ANALYSIS 

The linear regression equation is 


y = 0.02833 + 0.9928x 

where 

y = arc sin/CAMS rework estimated proportion 
- arc sin/true proportion 


and 

x = arc sin/CAMS initial estimated proportion 
- arc sin/true proportion 

\ 


The ANOVA for regression is shown in the following table. 


Source 

Degrees of 
freedom 

Sum of 
squares 

Mean ' 
square 
error 

F- ratio 

Regression 

1 . 

0.278 

0.278 

a 26 . 5 

Residual 

34 

.357 

.010 



Significant at 1-percent level of significance. 
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Thus, R 2 = J’JJI ~ 0.44 


3.3 ANALYSIS OF VARIANCE 

The ANOVA in this section is performed to study the site effect 
and the AI/DPA effect on the classification proportion errors. 


Source 

Degrees of 
freedom 

Sum of 
squares 

Mean 

square 

error 

F-ratio 

Site 

8 

0.12972 

0.01622 

a l . 954 

AI/DPA 

3 

.02817 

.00939 

1.131 

Residual 

24 

.19908 

.0083 


Total 

35 

.35698 




Significant at 10-percent level of significance. 

4. CONCLUSION 

The following conclusions are made with respect to. the ITS's 

selected for the present CAMS rework experiment. 

• CAMS rework performance is significantly correlated with CAMS 
initial performance in this test procedure. However, only 

44 percent of the error variability in CAMS rework results is 
caused by the CAMS initial errors. This implies that CAMS 
rework performance is as variable as the CAMS initial classi- 
fication performance. 

• No significant difference exists between performances of the 
two procedures, which implies that the CAMS rework procedures 
Offer no significant improvement. 

• No significant difference is apparent in terms of different 
AI/DPA Teams, which implies that the CAMS is repeatable for 
reviewing and reworking a segment. 
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• The effect of sites on the CAMS rework performance is statis- 
tically significant but only at the 10-percent level. 


a 
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TABLE I.- CAMS REWORKED SMALL-GRAIN PROPORTIONS, PHASE I ITS 
= Average of team proportions minus ground-truth proportions] 


Segment 

Biophases 

AI/DPA Team 

A 

Ground 

A 

B 

C 

D 

truth 

1687 

1,3 

35.2 

38.8 

33.0 

(a) 

5.9 

29.8 

1960 

1,3 

b 19 . 7 

b 19 . 7 

b 19 . 7 

b 19 . 7 

-11.7 

C 31 . 4 

1962 

1,2 

67.3 

70.0 

d 42.9 

d 27.7 

-9.8 

C 61. 8 

1963 

1,2 

31.7 

29.5 

d 28.1 

d 16.2 

-15.0 

e 41. 4 

1965 

1,2 

d 32.5 

d 32 . 5 

d 30.6 

d 32 . 5 

-20.3 

C 52.3 

1969 

1 , 2 , 3 , 4 

48.3 

45.4 

60.3 

33.1 

5.2 

42.8 

1970 

1,2,4 

32.0 

58.4 

62.7 

58.0 

11.3 

e 41.5 

1978 

1,3 

d 25.6 

d 25 . 6 

d 25.6 

d 32 . 6 

-14.6 

f 41. 9 

1979 

1,3 

30.9 

30.9 

30.9 

30.9 

-.2 

f 31.. 1 

1980 

1,3 

56.9 

54.3 

58.4 

66.5 

-1.3 

60.3 

1986 

l, g 2, g 3 

19.6 

24.4 

(a) 

(a) 

15.2 

6.8 


Overall 


AI/DPA Team declared segment unsuitable for classification. 
^Missing signatures. 

c Could be in error; no complete ground truth was available for 
verification, 
cl 

mislabeling. 

e Contains narrow strip fields. 

f Was recomputed to exclude the volunteer wheat. 

^Extremely hazy data. 
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TABLE II.- ACQUISITIONS FOR CAMS REWORK EXPERIMENT 


Segment 

Acquisition number for biophase 

1 

2 

3 

4 

1687 

74133 


75205 


1960 

74291 


75150 


1962 

74324 

75131 



1963 

74289 

75131 



1965 

75155 

75191 



a 1967 





1969 

75161 

75179 

75215 

75233 

1970 

75142 

75179 


75233 

1978 

74291 

! 

75133 


1979 

74291 


75133 


1980 

74291 


75133 


b 1986 



| 



^ot suitable for processing because of lack of ground 
truth . 

°Not suitable for processing because of haze and clouds. 










TABLE III,- ERROR SUMMARY FOR ITS REWORK 

error, where p = estimated percentage of wheat and 
p = ground- truth percentage of wheat] 


Segment 


AI/DPA Team 


1960 
19 62 
1963 
1965 

1969 

1970 

1978 

1979 

1980 


- 11.7 


- 9.7 

- 19.8 


- 3.5 

- 16.3 


- 3.4 


16.9 

- 16.3 


- 6.0 


11.7 

34.1 


Team average 

Team standard deviation 


- 6.12 
* 12 .99 


- 9.18 

9.11 


This column lists the original results from Team 0, 
which classified the segment during the Four-AI Experi- 
ment. The AI was randomly chosen from the four AI's 
and turned out to be AI Y. These results were from the 
latest results analyzed during the year by AI Y and were 
used as the starting point for the CAMS rework analysis. 

For AI/DPA Teams A through D. 
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TABLE IV.- COMPARISON OF CAMS REGULAR VERSUS ITS REWORK RESULTS 

" I = Improved result 
W = Worse than original 
N = Original accepted 
U = Segment declared unworkable 


Segment 

AI/DPA Team 

A 

B 

C 

D 

1687 

I 

W 

I 

u 

1960 

N 

N 

N 

N 

1962 

I 

I 

N 

W 

1963 

I 

I 

N 

W 

1965 

N 

N 

W 

N 

1969 

N 

I 

W 

I 

1970 

N 

W 

W 

W 

1978 

N 

N 

N 

I 

1979 

N 

N 

N 

N 

1980 

N 

W 

I 

W 

1986 

I 

I 

U 

U 

Totals 

12 I's 

3 U's 

10 W's 

19 N's 


